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Key findings 




Exploratory analyses showed that school-level student 
achievement growth during the 2012/13 school year in New 
Orleans open-enrollment public charter schools was positively 
associated with having: 

Kindergarten as an entry grade. 

An extended school year. 

More experienced teachers. 

It was negatively associated with having: 

A higher percentage of teachers with a graduate degree. 

A higher student-teacher ratio. 

More student supports offered. 
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Summary 


This exploratory study examined the associations of certain organizational, operation- 
al, and instructional features (“potential indicators”) of open-enrollment public charter 
schools in New Orleans, Louisiana, with school-level value-added measures of student 
achievement growth in the 2012/13 school year. The achievement domains included were 
English language arts, math, and science. 

Of the 85 public schools in the city during the 2012/13 school year, 75 were charter schools, 
supporting 92 different campuses and enrolling more than 84 percent of public school 
students in New Orleans. This predominance of charter schools makes New Orleans a 
particularly promising location for exploring the potential indicators of charter school 
effectiveness. 

Six features emerged from this study as potential indicators of New Orleans charter school 
effectiveness, particularly for school-level measures of student achievement growth in 
English language arts. School-level student achievement growth was positively associated 
with having kindergarten as an entry grade, an extended school year, and more experi¬ 
enced teachers. It was negatively associated with having a higher percentage of teachers 
with a graduate degree, a higher student-teacher ratio, and more student supports offered. 

This exploratory descriptive analysis is limited in important ways. Because it is non- 
experimental, the associations identified should not be interpreted as causal. Because the 
study had to be conducted at the school level and only 50 schools were in the potential 
indicator analysis sample, the study had a limited ability to identify systematic associations 
between school organizational, operational, and instructional features and school-level 
value-added measures of student achievement growth. 

Despite these limitations, the exploratory results are a first step in identifying charter 
school conditions that could be studied through more rigorous research to determine 
whether they are valid indicators of student success. 
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Why this study? 


For decades education policymakers and practitioners have been concerned about improv- 
ing student achievement in New Orleans, Louisiana. After the devastation of Hurricane 
Katrina in 2005, the public school system in New Orleans, already one of the lowest per- 
forming in the United States, was reorganized, and most schools became charter schools 
(Harris & Larsen, 2016). As a result, the number of charter schools in New Orleans 
expanded rapidly. Charter schooling is now the primary governance structure for public 
education in the city, making New Orleans unique among U.S. school districts. During 
the 2012/13 school year (the period covered by this study), 75 of the 85 public schools in 
New Orleans were chartered, enrolling more than 84 percent of public school students in 
the city in 92 school campuses. 

Public charter schools operate outside the traditional public school district system accord- 
ing to provisions of their organizational charter. The charter typically sets performance 
goals for the school but leaves school leaders free to determine the best means to achieve 
those goals. This arrangement provides charter schools with the flexibility to innovate in 
both their features and their operation. Therefore, charter schools provide an opportuni¬ 
ty to explore what school characteristics are potential indicators of school effectiveness 
(Maranto, Milliman, Hess, & Gresham, 2001). 

The Louisiana Charter Schools Research Alliance of the Regional Educational Laborato¬ 
ry (REL) Southwest requested an exploratory analysis of the distinctive features and poten¬ 
tial indicators of charter school effectiveness in New Orleans. The alliance was organized 
as a source of information on projects and studies that would be useful to charter school 
operators and overseers in the state. 1 


This study explored 
the association 
between various 
organizational, 
operational, and 
instructional 
features of charter 
schools in New 
Orleans and 
their students’ 
achievement 
growth 


Structural features of schools, such as entry grade and grade span, long have been thought 
to affect student achievement (for example, Purkey & Smith 1983), by either facilitating or 
hindering student learning. This study explored the association between various organiza¬ 
tional, operational, and instructional features of charter schools in New Orleans and their 
students’ achievement growth. Charter school features correlated with student achieve¬ 
ment gains are described in this study as “potential indicators” of charter school effective¬ 
ness. This study builds on prior research identifying a variety of school features that are 
potential indicators of charter school effectiveness (box 1). 
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Box 1. Potential indicators of charter school effectiveness identified in the 
research and used in the current study 

A small number of studies of charter schools have examined school characteristics that are 
potential indicators, or correlates, of charter school effectiveness. The following list of poten¬ 
tial indicators of effectiveness, identified in prior studies, includes characteristics for which 
reliable measures were available for the current study’s analysis: 

• Authorizer type (government entity or nonprofit organization; Witte, Weimer, Shober, & 
Schlomer, 2007; Zimmer, Gill, Attridge, & Obenauf, 2014). 

• School age (Bifulco & Ladd, 2006; Buddin & Zimmer, 2005; Hanushek, Kain, Rivkin, & 
Branch, 2007; Kelly & Loveless, 2012; Zimmer, Blanc, Gill, & Christman, 2008). 

• School size (Berends, Goldring, Stein, & Cravens, 2010; Gleason et al., 2010). 

• Grades served, including number of grades and kindergarten as the entry grade (Abdulkad- 
iroglu et al., 2009; Angrist, Cohodes, Dynarski, Pathak, & Walters, 2013; Jacob & Rockoff, 
2011; Zimmer et al., 2009). 

• Extended learning day (Angrist, Pathak, & Walters, 2011; Dobbie & Fryer, 2011; Gleason, 
Tuttle, Gill, Nichols-Barrer, & Teh, 2014; Macey, Decker, & Eckes, 2009; Musher et al., 
2005). 

• Extended school year (Hoxby, Murarka, & Kang, 2009; Musher, Musher, Graviss, & Strud- 
ler, 2005). 

• Percentage of teachers traditionally certified (Steele, Vernez, Gottfried, & Schwam-Baird, 
2011 ). 

• Teacher experience in years (Steele et al., 2011). 

Other studies of traditional public schools rather than charter schools have suggested 
that certain school features might be potential indicators of effectiveness. Those for which 
reliable measures were available for the current study’s analysis include: 

• Percentage of teachers traditionally certified (Darling-Hammond, Holtzman, Gatlin, & 
Heilig, 2005). 

• Percentage of teachers with a graduate degree (Betts, Zau, & Rice, 2003; Dee, 2004). 

• Teacher experience in years (Staiger & Rockoff, 2010). 

• Teacher quality, represented in this analysis by the percentage of teachers rated highly 
effective (Rockoff, 2004; Rivkin, Hanushek & Kain, 2005). 

• Class size, represented in this analysis by student-teacher ratio (Mosteller, 1997). 

• Availability of student support staff, such as counselors, speech therapists, and occu¬ 
pational therapists (Angrist & Lavy, 1999; Black & Wiliam, 1998; Ritter, 2000; Rockoff, 
2004; Rumberger & Palardy, 2005; Taylor & Tyler, 2012). 

RAND researchers, in partnership with the Cowen Institute for Public Education Initiatives 
at Tulane University, conducted an exploratory study of the features that distinguish New 
Orleans public charter schools from traditional public schools in the city (Steele et al., 2011). 
Two features identified by that study are included in the current study’s analysis: 

• Higher percentage of traditionally certified teachers. 

• Teachers with less experience. 

Additional information on each of these potential indicators of charter school effective¬ 
ness is in appendix A. The review process that yielded the list is described in appendix B. 
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What the study examined 


This exploratory analysis addressed the following research question: 

• What organizational, operational, and instructional features of New Orleans 
charter schools serving grades 3-8 are potential indicators of student achievement 
growth in English language arts, math, and science? 

The analysis focused on three types of potential indicators: organizational features, open 
ational features, and instructional features. Organizational features are related to a school 
as an organization, irrespective of its mission. Such features include age, size, scope, and 
source of authority and oversight, features that are relevant to all organizations regardless 
of what they do. Organizational features can be recognized by their inclusion in analyses 
of both education and noneducation organizations (for example, Downs 1967; Hoxby et 
al., 2009; Wolf, 1993). Operational features are related to the logistics of how a school 
functions. Such features include the length of the school day and school year as well as 
whether daily transportation is provided to students. Instructional features are related 
to how a school implements its mission of educating students. Such features, specific to 
schools, include classroom size, student supports, and the qualifications of teachers. 

These features were selected because prior research found them to be associated with 
student achievement growth (see box 1) and because policymakers and practitioners have 
at least some ability to shape them at the school level. (Comparisons between the popula- 
tion of New Orleans public charter schools and the entire population of Louisiana public 
schools on these and other features can be found in appendix C.) 

New Orleans public charter schools tend to serve a distinct population of Louisiana stu- 
dents and typically do so in ways that differ from the ways in which traditional public 
schools do so. Charter school students in New Orleans are more likely than traditional 
public school students in Louisiana to be Black and to come from low-income households 
but are less likely to be Hispanic. On average, New Orleans charter schools have wider 
grade spans than traditional public schools in Louisiana and are likely to include both kin- 
dergarten and grade 6, and sometimes grade 12 as well, suggesting that the charter school 
model is to enroll students for an extended number of years before transitioning them to 
another school. New Orleans charter schools have fewer student supports, suggesting that 
they rely more on classroom teachers in addressing the needs of disadvantaged students 
than do traditional public schools in Louisiana. The proportion of teachers rated highly 
effective in New Orleans charter schools is lower than the statewide average. 

The analysis focused on the extent to which the selected features are correlated with the 
schooHevel test-score gains of elementary and middle school students in New Orleans 
charter schools. 2 Key terms used in this report are described in box 2. A summary of data 
sources and analysis methods is in box 3. 


New Orleans 
public charter 
schools tend to 
serve a distinct 
population of 
Louisiana students 
and typically do 
so in ways that 
differ from the 
ways in which 
traditional public 
schools do so 
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Box 2. Key terms 


Charter school. A public school authorized to operate with some degree of independence from 
district or state public school regulations while being held accountable for student outcomes 
by an authorizing entity. This study counts each school campus of a charter school operator 
within the geographic boundary of New Orleans as an individual New Orleans public charter 
school, because the Louisiana Department of Education assigns each school campus a unique 
school identification number. 

Indicator. A feature of a school that is statistically associated with school-level outcomes, in 
this case student achievement growth in English language arts, math, or science (value-added 
measure scores). Features of schools that research has indicated might explain school effec¬ 
tiveness and that can be altered by policymakers or practitioners are potential indicators. This 
study examines potential indicators of effectiveness among New Orleans charter schools. 

Open enrollment. A type of public charter school that does not apply any restrictive admis¬ 
sions standards on students. Such schools admit all interested applicants to the grades that 
they serve and enroll students by lottery if they have more applicants than seats. Language 
immersion schools that required applicants to have some previous exposure to the language 
were classified as open enrollment because such a minimum requirement is not likely to be 
restrictive for the subpopulation of students applying to such schools. Schools that required 
applicants to score above a certain level on a general achievement test were classified as 
selective-admissions schools rather than open-enrollment schools because such a test is 
likely to be restrictive for the subpopulation of students applying. 

School authorizer. A legal entity with the responsibility to decide whether new charter schools 
should open, to monitor charter school operation and performance, and to decide whether 
existing charter schools should close. Charter school authorizers in Louisiana include public 
school boards, the Recovery School District, and the Louisiana Board of Elementary and Sec¬ 
ondary Education. 

Student support staff index. Count of how many of the following student supports are offered 
at the school: speech therapy, occupational therapy, adapted physical education, mental 
health coordination, on-site counseling, school psychologist, mentoring, and peer supports. 

Value-added measure. Average learning gains that could reasonably be attributed to stu¬ 
dents’ education experiences in school. Value-added measures were calculated by the Loui¬ 
siana Department of Education for English language arts, math, and science. This study uses 
school-level value-added measure scores provided by the Louisiana Department of Education. 
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Box 3. Data sources and research methods 


Data 

This study used school-level data provided by the Louisiana Department of Education and by 
the New Orleans Parents’ Guide, a nonprofit community organization that collects information 
from all public schools in the city and produces the annual New Orleans Parents’ Guide to Public 
Schools (New Orleans Parents’ Guide, 2013; see appendix B for more information). 

The Louisiana Department of Education data included: 

• School-level value-added measure scores in English language arts, math, and science that 
measure student achievement growth in scale score units while controlling for student 
background characteristics. 

• Measures of the following potential indicators: school age, size, and number and range of 
grades served; inclusion of prekindergarten, kindergarten, grade 6, and grade 12; school 
authorizer type; percentage of teachers certified; teacher experience; student-teacher 
ratio; and percentage of teachers rated highly effective in practice. 

• Measures of the characteristics of the student population served by New Orleans charter 
schools. These measures are listed in appendix B for descriptive purposes, but they were 
not used in the analysis. These descriptive variables included percentage of students from 
low-income households (identified by eligibility for the federal school lunch program), per¬ 
centage of Black and Hispanic students, percentage of English learner students, percent¬ 
age of female students, and percentage of students in special education. 

The New Orleans Parents’ Guide (2013) data covered the potential indicator variables of 
schools’ use of an extended school day or year, student supports offered, and the provision of 
transportation or tutoring. 

Sample 

This report presents analytic results for a sample of open-enrollment New Orleans public charter 
schools operating during the 2012/13 school year that enrolled a sufficient number of students 
in tested grades to produce at least one school-level student achievement growth score from 
the spring 2013 test administration (see descriptive information about the sample in table B2 
in appendix B). The analytic sample consisted of 49 or 50 of the 92 charter school campuses, 
depending on the subject area with student achievement growth scores (see table B2). Seven 
charter school campuses were excluded because they had exclusionary admissions requirements, 
and thirty-five were excluded because they lacked student achievement growth scores. Descrip¬ 
tive statistics are provided in appendix B for the entire population of 92 charter school campuses, 
but only the analytic sample of 49 or 50 campuses was used in the potential indicator analysis. 

Methodology 

The potential indicator analysis used ordinary least-squares regression to estimate the extent to 
which each potential indicator variable was systematically associated (covaried) with variation in 
school-level student achievement growth scores, while the association between all other potential 
indicators and school-level student achievement growth scores was controlled for. An indicator 
variable for whether the school offered prekindergarten was included as a control variable in the 
regressions so that the effect of kindergarten alone would be clear. Associations are judged to be 
statistically significant if there is a low likelihood that the association is due to random chance or 
statistical noise. The standard threshold for statistical significance in education evaluations of p < 
.05 was used in this analysis. Only school features whose association with a student achievement 
growth outcome was statistically significant were considered potential indicators of charter school 
effectiveness. See appendix B for additional information on study methods. 
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What the study found 


Regression analysis found that several New Orleans public charter school features were sta- 
tistically significantly associated with schooUevel student achievement growth, especially 
in English language arts achievement (table 1). Those features are considered potential 
indicators of New Orleans charter school effectiveness. 


The organizational feature of kindergarten as an entry grade, a subindicator of grades 
served, was statistically significantly associated with higher student achievement growth in 
English language arts. The operational feature of an extended school year also was statisti¬ 
cally significantly associated with higher student achievement growth in English language 
arts. The instructional features of a lower percentage of teachers with a graduate degree, 
more experienced teachers, and a lower student-teacher ratio were statistically significant¬ 
ly associated with higher student achievement growth in English language arts (see table 
D1 in appendix D for detailed statistical results). 

Two organizational school features had associations with English language arts student 
achievement growth that were statistically significant at the 90 percent confidence level 


Table 1. Association of potential indicators of effectiveness of New Orleans 
public charter schools with English language arts, math, and science value-added 
measure scores, 2012/13 (unstandardized regression coefficient estimates) 


Feature 

English 
language arts 

Math 

Science 

Organizational 

Authorized by Recovery School District 3 

3.78 

4.65 

2.85 

School age 

-0.78 

-0.78 

0.13 

School size in number of students 

1.48 

2.08 

0.08 

Number of grades served 

-1.83 

-2.21 

0.59 

Includes prekindergarten 

-4.51 

-7.06 

-2.12 

Includes kindergarten 

15.30** 

22.59* 

0.31 

Operational 

Offers extended learning day 

-3.11 

-0.95 

-2.42 

Offers extended school year 

5.22* 

6.45 

3.09 

Instructional 

Percentage of teachers traditionally certified 

12.20 

11.15 

-0.74 

Percentage of teachers with graduate degree 

-25.55* 

-33.83 

-4.49 

Teacher experience in years 

1.09* 

0.98 

0.08 

Percentage of teachers rated highly effective 

2.87 

1.15 

-1.86 

Student-teacher ratio 

-0.92* 

-0.82 

-1.24* 

Student support staff index b 

-0.15 

-0.90 

-2.00* 


* Significant at p < .05; ** significant at p < .01 

Note: Value-added measure scores are in scale score units. F-statistic for English language arts model = 3.10; 
probability > F = .00. F-statistic for math model = 1.09; probability > F = .41. F-statistic for science model = 1.27; 
probability > F = .28. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 
of chronically low-performing schools. 

b. Count of each of the student supports offered at a school. 

Source: Authors' calculations based on school-level data provided by the Louisiana Department of Education 
and New Orleans Parents’ Guide (2013). 


Kindergarten as 
an entry grade, an 
extended school 
year, a lower 
percentage of 
teachers with a 
graduate degree, 
more experienced 
teachers, and a 
lower student- 
teacher ratio were 
all statistically 
significantly 
associated with 
higher student 
achievement 
growth in English 
language arts 
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but fell short of statistical significance at the 95 percent standard used in this study: larger 
school size was associated with higher student achievement growth in English language 
arts, and more grades served was associated with lower student achievement growth in 
English language arts. These results should be interpreted with caution because they fail to 
meet the conventional standard for statistical significance. 

The analysis revealed fewer potential indicators of charter school effectiveness associated 
with student achievement growth in math and science than in English language arts (see 
tables D1-D3 in appendix D for detailed statistical results). Having kindergarten as an 
entry grade was the only school feature that had a statistically significant association with 
higher student achievement growth in math. Having a lower student-teacher ratio and 
fewer student supports offered were the only school features that had a statistically signifi- 
cant association with higher student achievement growth in science. 

The features of New Orleans charter schools in 2012/13 had no statistically significant 
associations with student achievement growth in 2013/14. This could be due to the original 
findings being false discoveries of associations by mere chance, misalignment of 2012/13 
measures of the potential indicator variables with the 2013/14 student achievement growth 
outcome data, or less precision in the 2013/14 value-added measures of student achieve¬ 
ment growth. The analysis could not distinguish among these three possible explanations 
for the nonrobustness of the findings across outcome years. Specifically, the regression 
coefficients for the eight cases of associations that were statistically significant for the main 
analysis on the 2012/13 outcome data were all smaller when the analysis was repeated on 
the 2013/14 outcome data, consistent with both false discoveries and misalignment as pos¬ 
sible explanations for the differences. The standard errors of the regression estimates from 
the repeat of the analysis on the 2013/14 outcome data were larger than those from the 
main analysis on the 2012/13 outcome data in every case except for one, consistent with 
the possible explanation that less precise student achievement growth measures in 2013/14 
explain the discrepancy. 

Having kindergarten as an entry grade and having a lower student-teacher ratio were the 
only potential indicators that had a statistically significant association with more than one 
student achievement growth outcome. Having kindergarten as an entry grade was associ¬ 
ated with higher student achievement growth in English language arts and math. Having 
a lower student-teacher ratio was associated with higher student achievement growth in 
English language arts and science. 

Implications of the study findings 


Having 

kindergarten as an 
entry grade was 
the only school 
feature that had 
a statistically 
significant 
association with 
higher student 
achievement 
growth in math, 
and having a lower 
student-teacher 
ratio and fewer 
student supports 
offered were 
the only school 
features that had 
a statistically 
significant 
association with 
higher student 
achievement 
growth in science 


This study revealed statistically significant associations between student achievement 
growth (especially in English language arts) and six New Orleans public charter school 
features in 2012/13 that may be used as potential indicators of charter school effectiveness 
in the future. Charter schools that had kindergarten as an entry grade, an extended school 
year, and more experienced teachers tended to have higher student achievement growth 
scores. Charter schools with a higher proportion of teachers with a graduate degree, higher 
student-teacher ratios, and more student supports offered tended to have lower student 
achievement growth scores. 
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One exploratory finding from the current study was inconsistent with the prior literature. 
Having more student supports was associated with lower achievement, which contrasts 
with previous research findings that having more support staff was associated with higher 
student test scores (Angrist & Lavy, 1999; Black & Wiliam, 1998; Ritter, 2000; Rockoff, 
2004; Rumberger & Palardy, 2005; Taylor & Tyler, 2012). The exploratory results in the 
current study are correlational and may be due largely to variables that could not be 
included in the analyses rather than to causal connections. It also is possible that student 
outcomes are influencing potential indicators and not vice-versa. For example, persisting 
low student achievement may have led some schools to offer more supports to help strug¬ 
gling students. 

Other exploratory findings of the current study were consistent with prior research on 
school effectiveness, either in general or specifically for public charter schools. For example, 
previous studies have found that years of teacher experience are predictive of larger student 
test score gains (Staiger & Rockoff, 2010; Steele et al, 2011). Another study found that 
an extended school year was associated with greater charter school effectiveness in New 
York City (Hoxby et al., 2009), as was the case in the current study for charter schools in 
New Orleans. And other studies using more precise classroom-by-classroom counts of stu¬ 
dents per teacher than the overall student-teacher ratio in each school used in the current 
study have also found that a smaller class size is associated with higher test score gains 
(Mosteller, 1997). 

These exploratory results identify features that should be tested with more rigorous 
research methods before they are used in making decisions about the design and operation 
of public charter schools. 

More potential indicators of charter school effectiveness were identified in the analysis 
of associations with student achievement growth in English language arts (five potential 
indicators) than in analyses of associations with student achievement growth in math 
(one potential indicator) and science (two potential indicators). Four potential indicators 
have a statistically significant association with student achievement growth in English lan¬ 
guage arts but not student achievement growth in math, even though the math coefficient 
estimates are larger (for having an extended school year and the percentage of teachers 
with a graduate degree) or very similar in size (for teacher experience and student-teacher 
ratio), suggesting—but not proving—that the measures of student achievement growth 
may have been more precise for English language arts than for math. Since the reasons 
for the discrepancy are not certain, additional studies that explore associations between 
potential indicators of charter school effectiveness and student achievement growth in 
all three domains (English language arts, math, and science) would be helpful in deter¬ 
mining whether this pattern is systematic or limited to the case examined here. Research 
on potential indicators that can replicate results across multiple years of student achieve¬ 
ment growth also would be welcome, since this study’s findings were not consistent across 
outcome years. 

Research has identified multiple potential indicators of charter school effectiveness (see 
appendix A). Thus, it would be helpful if schools collected and reported a rich set of infor¬ 
mation about these potential indicators, such as the amount and difficulty of assigned 
homework and formative assessment practices, on a regular basis to facilitate a more com¬ 
prehensive assessment. 


The finding that 
having more 
student supports 
was associated 
with lower 
achievement 
contrasts with 
previous research 
findings that 
having more 
support staff was 
associated with 
higher student 
test scores 


8 


Limitations of the study 


The analysis was limited by the potential indicator variables for which data were available. 
Some indicators, including transportation and tutoring, were almost universal features 
among New Orleans charter schools so that they could not be used to distinguish more 
effective schools from less effective schools. Data were not available on other measures 
of potential indicators, such as universally high expectations for students, the quality of 
principal leadership, and the amount of parent involvement in a child’s education. Analyz- 
ing these potential indicators would have required administering surveys to teachers and 
students or parents, original data collection activities that were beyond the scope of this 
study. 


Because the study relied on school-level data, the sample size for the regression analysis 
was small (49 or 50 observations). It is possible that some potential indicators of charter 
school effectiveness were not identified because of low study power. A direct analysis of 
student'level data would have permitted a more fine-grained and higher-powered explo¬ 
ration of the potential indicators of New Orleans charter school effectiveness. However, 
recently enacted student privacy laws in Louisiana prevented the Louisiana Department 
of Education from releasing student-level achievement data to the study team. Instead, 
the department used student-level achievement data to generate the school-level student 
achievement growth scores that were then shared with the study team. 

The data for this study were limited to the 2012/13 and 2013/14 school years, with the 
main analysis focused on the 2012/13 school year, when the potential indicator measures 
and student achievement outcome measures were aligned. It is possible that some of the 
findings were particular to conditions that students experienced that year and might not 
hold in other years. A longitudinal study with multiple years of data on student achieve¬ 
ment growth outcomes and indicator data aligned with those outcomes would produce 
more reliable estimates of potential indicators of charter school effectiveness. Such a study 
was not possible in this case, so readers are cautioned not to draw firm conclusions from 
these findings. 


Because the study 
relied on school- 
level data, the 
sample size for the 
regression analysis 
was small (49 or 
50 observations). 

It is possible that 
some potential 
indicators of 
charter school 
effectiveness were 
not identified 
because of low 
study power 


All the variables examined likely contained some degree of measurement error (Kmenta 
1986). In this study the data for the dependent variables were provided by the Louisiana 
Department of Education, which calculated the student achievement growth scores for 
individual schools based on aggregate student performance on the Louisiana Educational 
Assessment Program and Integrated Louisiana Assessment Program accountability tests. 
These are typical criterion-referenced tests designed by psychometrics consultants working 
with Louisiana educators and government officials (Louisiana Department of Education, 
2007). The procedure used to generate school-level student achievement growth scores was 
consistent with professional practice and drew on data for student prior-year achievement 
and background characteristics (see appendix B). Still, some degree of measurement error 
is likely since the dependent variables used in this study were themselves the product of 
statistical estimation. Staff of the Louisiana Department of Education suggested to the 
study team that they had more confidence in the reliability of the student achievement 
growth estimations for English language arts and math than in the reliability of the student 
achievement growth estimations for science. Student achievement growth estimations 
for English language arts and math count double in the Louisiana school accountability 
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metric, so school personnel have an incentive to focus on producing learning gains in 
those areas. 

For the potential indicators used in the analysis, the magnitude of error is likely of an 
acceptable level given the involvement of the Louisiana Department of Education in data 
collection quality control for all the variables in the study, including those from the New 
Orleans Parents’ Guide to Public Schools (New Orleans Parents’ Guide, 2013). Potential 
indicators of charter school effectiveness that likely are not subject to precise measure- 
ment, such as disciplinary policy, were excluded from the study at the planning stage due 
to concerns about severe measurement error. 

The student achievement growth that informed the potential indicator analysis excluded 
certain important student subgroups. Students in grades K-3 and 9-12 in 2013, students 
who changed schools in 2013, and students with disabilities who took an alternative assess¬ 
ment in place of the standard state test were not represented in the baseline and outcome 
tests that generated the school-level student achievement growth calculations. The results 
of the potential indicator portion of this study apply directly only to the population of 
stable New Orleans charter school students in a regular education program in grades 4-8 
in 2012/13. 

The study is limited to the New Orleans open-enrollment charter schools with 2012/13 
student achievement growth scores generated from 2011/12 and 2012/13 accountabili¬ 
ty testing. Schools that did not operate in both 2011/12 and 2012/13, because they were 
new or because they closed in 2012/13, were excluded from the sample, as were schools 
that required an academic entrance exam and schools that enrolled too few students in 
tested grades to report school-level scores without violating privacy laws. These schools 
could not be included in the analytic sample because their selective nature would have 
made them too different from the other charter schools to include (in the case of schools 
with entrance exams) or because there were no data for them (in the case of schools that 
enrolled too few students to report scores). Thus, the results of this study do not necessary 
apply to these types of New Orleans charter schools. 

Many of the potential indicators analyzed are not fully independent of each other. For 
example, class size, school size, and school grade span all will be strongly correlated with 
each other in charter schools with just one classroom per grade. Such multicollinearity 
among explanatory variables in regression estimations does not generate bias but does 
increase the random error of the estimates (Kmenta, 1986). Since the analysis already is 
underpowered, inefficiency due to a high degree of multicollinearity among some of the 
potential indicators is an added limitation of the study. 

Given the limitations of this exploratory study, practitioners are cautioned to consider 
the findings as only a first step in identifying potential indicators of school effectiveness. 
The most important function of these findings is to signal to researchers which features 
of public charter schools should be evaluated more rigorously. Should follow-up studies 
confirm that certain school features are causes of public charter effectiveness rather than 
merely correlates, practitioners would then be well advised to adopt those organizational, 
operational, or instructional features in their schools. 


The results of 
the potential 
indicator portion 
of this study apply 
directly only to 
the population 
of stable New 
Orleans charter 
school students in 
a regular education 
program in grades 
4-8 in 2012/13 
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Appendix A. Literature review 


This appendix summarizes the research literature on potential indicators of charter school 
effectiveness and identifies which indicators could not be included in the research. It also 
discusses previous research on education in New Orleans and the role of public charter 
schools in the city. 

Potential indicators of student achievement 

A systematic literature review identified 23 potential indicators of charter school student 
achievement. Twelve of these indicators were included in the potential indicator analysis 
in this study, either through indicator or subindicator variables; eleven were not. 

The following indicators were included in the analysis: 

1. Authorizer type. 

2. School age. 

3. School size. 

4- Grades served, represented in the analysis by the subindicators of number of grades 
served and inclusion of kindergarten as an entry grade. 

5. Extended learning day. 

6. Extended school year. 

7. Percentage of teachers traditionally certified 

8. Percentage of teachers with a graduate degree. 

9. Teacher experience in years. 

10. Teacher quality, represented in the analysis by the percentage of teachers rated 
highly effective. 

11. Class size, represented in the analysis by student-teacher ratio. 

12. Availability of student support staff (such as counselors, speech therapists, and 
occupational therapists). 

The following indicators were not included in the analysis: 

1. Availability of tutoring. 

2. Knowledge is Power Program (KIPP) school model. 

3. For-profit or nonprofit status. 

4- Administrative structure of the charter management organization. 

5. Academic press. 

6. Extent of instructional innovation. 

7. Principal leadership. 

8. Parent involvement. 

9. Disciplinary policy. 

10. Amount and difficulty of homework. 

11. Use of formative assessment. 

Potential indicators of student achievement in charter schools that were included in the analysis 

1. Authorizer type. Two studies have found that charter schools authorized by government 
entities tend to produce higher achievement growth than do charter schools authorized by 
independent nonprofit organizations (Witte et al., 2007; Zimmer et al., 2014). New Orleans 
charter schools are all authorized by government agencies: the Recovery School District (a 
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special school district run by the Louisiana Department of Education that intervenes in 
the management of chronically low-performing schools), the Orleans Parish School Board, 
or the Louisiana Board of Education. 

2. School age. School age is the most commonly reported potential indicator of charter 
school effectiveness. Mature charter schools produce higher student achievement growth 
(Bifulco & Ladd, 2006; Buddin & Zimmer, 2005; Cremata et al., 2013; Hanushek et al, 
2007; Kelly & Loveless, 2012; Zimmer et al., 2008). As with the association between teacher 
experience and student test scores, researchers hypothesize that older charter schools have 
higher student achievement growth both because less effective charter schools close down 
and because young charter schools learn how to operate more effectively over time. 

3. School size. All else equal, smaller public charter schools tend to perform more effec¬ 
tively (Berends et al, 2010; Gleason et al., 2010). Smaller schools are most clearly predictive 
of higher student achievement growth in the high school grades, when schools tend to be 
largest (Egalite & Kisida, 2016). Research suggests that smaller (but not extremely small) 
education communities are especially helpful in boosting the achievement of students 
from low-income households and racial/ethnic minority students (Lee & Smith, 1997). 

4. Qrades served. Three studies have found significantly higher student achievement 
growth in charter schools that serve a broader range of grades, especially if they include 
both junior high and high school grades, than in charter schools that serve a narrower 
range of grades (Abdulkadiroglu et al., 2009; Angrist et al., 2013; Zimmer et al., 2009). The 
researchers hypothesize that eliminating the transition from junior high to high school 
produces less social and education disruption and therefore enables greater continuity in 
learning. Elaving kindergarten as an entry grade also is hypothesized to improve student 
learning outcomes by enabling students to enter a school at kindergarten without the dis¬ 
ruption of transitioning from a previous school. Only 4 of the 50 public charter schools in 
the analytic sample for this study included grade ranges that spanned both junior high and 
high school grades, so the study was unable to explore the specific claim that combining 
those grade levels is a potential indicator of New Orleans charter school effectiveness. 
The subindicators of the number of grades served and kindergarten as an entry grade were 
included in the potential indicator analysis. 

5. Extended learning day. Several studies have reported that charter schools with added time 
for formal academic instruction, beyond the standard seven-hour school day, have a stron¬ 
ger positive effect on student achievement (Angrist et al, 2011; Gleason et al., 2014; Macey 
et al., 2009; Musher et al., 2005). The extra time needs to be structured and to have academic 
content for extended school days to be positively associated with student achievement. 

6. Extended school year. Increasing learning time by lengthening the school year also 
has been identified as a potential indicator of charter school effectiveness (Hoxby et al., 
2009; Musher et al., 2005). Distinct from year-round schooling models, which involve the 
same number of instruction days as conventional school calendars, extended school years 
add instruction days either in the summer or on Saturdays. Charter schools in New York 
City provide as many as 220 instruction days, compared with the typical 180-185 days 
in traditional public schools, and their extended school years have been identified as the 
most likely reason for charter schools outperforming traditional public schools in that city 
(Hoxby et al., 2009). 
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I. Percentage of teachers traditionally certified. California schools that rely less on 
alternatively licensed teachers have been found to produce higher achievement growth 
(Zimmer & Buddin, 2007). The effect was consistent across the samples of public charter 
and traditional public schools. 

8. Percentage of teachers with a graduate degree. Teachers with a graduate degree have 
been found to produce higher student achievement growth (Betts et al., 2003; Dee, 2004). 
However, findings have not been consistent across studies with different methodologies or 
samples (Buddin & Zamarro, 2009; Harris & Sass, 2011). 

9. Teacher experience in years. Studies have concluded that teachers become more effec¬ 
tive at increasing student achievement the longer they work as classroom teachers (Gre- 
enwald, Hedges & Lane, 1996; Staiger & Rockoff, 2010). Researchers suggest two reasons 
for the association between teacher experience and higher student achievement growth. 
First, administrators are able to observe young teachers and counsel out of teaching those 
who perform poorly (Staiger & Rockoff, 2010). Thus, teachers who continue past their for¬ 
mative years tend to be higher in the student performance distribution. Second, teachers 
learn by doing. Those who continue in the profession become more effective in boosting 
student performance, with a large share of the gains occurring in the first five years of their 
career (Staiger & Rockoff, 2010; Rivkin et al., 2005). Higher levels of teacher experience 
might be especially strongly associated with higher student performance growth in New 
Orleans public charter schools because charter school teachers in the city averaged 9.7 
years of experience compared with an average of 13 for traditional public school teachers 
in 2008/2009 (Steele et al., 2011). New Orleans charter schools with teachers that average 
more years of experience might be particularly effective, relative to peer charter schools 
with teachers that average less years of experience, because there are so few teachers with 
many years of experience in the charter school sector. 

10. Teacher quality. The quality of teachers has a strong association with student 
achievement growth (Rockoff, 2004; Rivkin et al., 2005). In this study, teacher quality was 
measured based on ratings of whether teachers were highly effective in the classroom by 
independent observers using a rubric developed by the Louisiana Department of Education. 

II. Class size. All else equal, smaller class size tends to be associated with higher student 
achievement growth, especially in early elementary grades (Mosteller, 1997). The effects of 
class size on student achievement growth appear to be highly context dependent, varying 
with the maximum class size established by policy, with implementation fidelity, and with 
scale (Bohrnstedt & Stecher, 2002). This study used teacher-student ratio as the measure 
of class size. 

12. Availability of student support staff (such as counselors, speech therapists, and 
occupational therapists). Schools with more specialized support staff tend to generate 
higher student achievement growth (Angrist & Lavy, 1999; Black & Wiliam, 1998; Ritter, 
2000; Rockoff, 2004; Rumberger & Palardy, 2005; Taylor & Tyler, 2012). Education spe¬ 
cialists, school counselors, and nurses all have been linked to higher student test scores in 
various contexts. 
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Potential indicators of student achievement in charter schools that were not included in the analysis 


Eleven potential indicators of charter school effectiveness identified in the literature could 
not be included in this study for the reasons given below. 

1. Availability of tutoring. Charter schools that provide tutoring to struggling students 
have been shown to increase student achievement (Dobbie & Fryer, 2011). The current 
study could not examine tutoring as a potential indicator of charter school effectiveness 
because all but five New Orleans charter schools in the analytic sample (90 percent) pro- 
vided in-school tutoring to their students. 

2. Knowledge is Power Program (KIPP) school model. KIPP schools are one of the 
best known national charter school networks. Researchers have found the KIPP educa- 
tion model to have a positive effect on student achievement (Macey et al., 2009; Angrist, 
Dynarski, Kane, Pathak, & Walters, 2010). The KIPP model could not be included in this 
analysis because too few New Orleans charter schools are KIPP schools. 

3. For-profit or nonprofit status. Research suggests that charter schools that are incorpo¬ 
rated as nonprofit organizations are more effective at raising student achievement than are 
charter schools incorporated as for-profit companies (King, 2007). The for-profit or non¬ 
profit status of schools could not be included in this analysis because too few New Orleans 
charter schools were for-profit. 

4. Administrative structure of the charter management organization. Charter manage¬ 
ment organizations (CMOs) are administrative units that provide back-office services and 
oversight to charter school franchises. The administrative structure of the CMO has been 
linked to higher student achievement in public charter schools (Furgeson et al., 2012). An 
indicator variable for administrative structure of the CMO could not be included in this 
analysis because information about that feature was not available from the databases used 
in the study. 

5. Academic press. Academic press is the extent to which school personnel emphasize aca¬ 
demic achievement in the culture and daily operations of the school. Academic press has 
been linked to higher test score performance in public charter schools (Berends et al., 2010; 
Cheng, Hitt, & Kisida, 2015). Academic press could not be included in this analysis because 
information about that feature was not available from the databases used in the study. 

6. Extent of instructional innovation. Instructional innovation involves adopting a novel 
or unconventional approach to delivering educational content. One previous charter 
school effectiveness study that examined whether instructional innovation was associated 
with learning gains in public charter schools found innovation measures to be negatively 
associated with gains, suggesting that the disruption from such changes might not be worth 
the benefits of adopting a new instructional approach. (Berends et al, 2010). Instructional 
innovation could not be included in the study because information about that feature was 
not available from the databases used in the study. 

7. Principal leadership. Charter school principals are the organizational and instruction¬ 
al leaders of their schools. A few charter school studies have evaluated the quality of prin¬ 
cipal leadership, either through teacher evaluations of their principals or through principal 
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self-reports of engaging or not engaging in a defined set of leadership practices (Camp¬ 
bell & Gross, 2008; Macey et al, 2009; Witte et al, 2007). Charter schools with higher 
ratings for principal leadership were more likely to outperform their local traditional public 
schools in promoting student achievement (Macey et al, 2009; Witte et al, 2007). Princi¬ 
pal leadership could not be included in this study because information about the feature 
was not available in the study databases. 

8. Parent involvement. Parents can be involved in their child’s education in a variety 
of ways, including volunteering at school and assisting with homework. Several charter 
school studies have examined the association between level of parent involvement and 
student achievement, finding that more involvement was associated with higher achieve¬ 
ment (Downing, Spencer, & Cavallaro, 2004; Zimmer & Buddin, 2007). Parent involve¬ 
ment could not be included in this study because information about the feature was not 
available in the study databases. 

9. Disciplinary policy. Disciplinary policies that do not exclude students from their school 
environment are associated with higher student achievement growth (Arcia, 2006; Beck & 
Muschkin, 2012; Skiba & Rausch, 2004). School disciplinary policy could not be included 
in this study because the measure of this feature available in the study databases was based 
on the descriptions of principals, raising concerns about precision and reliability. 

10. Amount and difficulty of homework. Assigning a greater amount of, and more chal¬ 
lenging, homework has been associated with higher student achievement growth (Wolf & 
Hoople, 2006). Amount and difficulty of homework could not be included in this study 
because information about the feature was not available in the study databases. 

11. Use of formative assessment. The frequent use of diagnostic tests to gauge student 
mastery of content, often called formative assessment, has been linked to higher student 
achievement growth in general (Wolf, 2007) and specifically for students in charter schools 
(Angrist et al., 2013; Chabrier, Cohodes, & Oreopoulos, 2016; Hoxby et al, 2009). The 
frequency of the use of formative assessment could not be included in this study because 
no information about this feature was available in the study databases. 

Charter school student achievement in New Orleans 

The post-Hurricane Katrina education landscape has resulted in an unprecedented prolif¬ 
eration of charter schools in New Orleans. Public charter schools constitute 88 percent of 
public schools in New Orleans and enroll 84 percent of the city’s public school students. 
In addition, 75 of the 104 public charter school campuses in Louisiana are in New Orleans 
(Louisiana Department of Education, 2013). 

Several studies have examined features of the New Orleans charter school landscape since 
Hurricane Katrina (for example, Kingsland & Donovan, 2013; Rasheed, 2006; Tillotson, 
2006). Harris (2015, p. 12), writing on behalf of an extensive Education Research Alliance 
team, reported that the replacement of the New Orleans traditional public school system 
with an overwhelmingly charter school alternative produced “an upward change in the 
trajectory of student test scores in New Orleans after the reforms of about 0.2 to 0.4 stan¬ 
dard deviations, enough to improve the typical student’s performance by 8 to 15 percentile 


points.” The study was not able to identify potential indicators of student achievement 
growth in New Orleans charter schools. 

Two Center for Research on Education Outcomes studies compared student achievement 
in New Orleans charter schools with the achievement of matched students in traditional 
public schools (Center for Research on Education Outcomes, 2009; Cremata et al., 2013). 
These studies reported that the typical charter school student in Louisiana outperformed 
his or her matched comparison peer in both English language arts and math, with effect 
sizes in the 2013 study of 0.07 standard deviation in English language arts (equivalent to 50 
days of learning) and 0.09 standard deviation in math (equivalent to 65 days of learning) 
and effect sizes in the 2009 study of 0.06 standard deviation in both English language arts 
and math. Because 72 percent of Louisiana charter schools in the 2013 study (Cremata 
et al., 2013) were in New Orleans, this statewide Louisiana charter school achievement 
effect is likely to reflect results specific to New Orleans as well. 

In partnership with the Cowen Institute at Tulane University, RAND conducted an 
exploratory study of the features of New Orleans public charter schools that distinguish 
them from traditional public schools in the city (Steele et al., 2011). New Orleans charter 
schools had principals who reported higher decisionmaking autonomy and more contract- 
ing-out for services, teachers who reported greater ease in maintaining discipline, school 
personnel who were less likely to say that the school faced serious challenges in promoting 
student achievement (that is, they evidenced a “no excuses” attitude to boosting student 
learning), a smaller percentage of teachers who were alternatively licensed, a larger per¬ 
centage of less-experienced teachers, less professional development for teachers, a higher 
percentage of parents who listed academic reasons for choosing the school as opposed to 
ease of transportation, and a larger percentage of satisfied parents. Not all of these indi¬ 
cators were included in the list at the beginning of this appendix because the Steele et al. 
study was limited by its nonexperimental research methodology and low survey response 
rates (52 percent for teachers, 36 percent for parents, and 32 percent for principals). 

Conclusion 

The literature examining student achievement in charter schools continues to grow. 
Several of the reviewed studies compared student achievement or achievement growth in 
charter schools and traditional public schools. Research that identifies causal relationships 
between indicators (or moderators) of charter school effectiveness and student outcomes is 
more limited. However, preliminary results from this literature review revealed 18 potential 
indicators that have been associated (or hypothesized to be associated) with differences 
in student outcomes in charter schools. Data were available to include 12 of the potential 
indicators in this exploratory study. 
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Appendix B. Literature review process, data, and methodology 


The first step in this exploration of the potential indicators of public charter school effec- 
tiveness in New Orleans was to review the research literature systematically to determine 
what is known about the school features that are associated with charter school effec- 
tiveness (see appendix A). Those previous findings then informed the study team’s data 
requests and analytic models. This appendix describes the data and methodology employed 
to conduct the literature review and the exploratory analysis that followed. 

Literature review 

The study team followed a structured protocol for searching academic databases for peer- 
reviewed journal articles or reports published between 2000 and 2014 on charter school 
performance and potential indicators of effectiveness. Initial searches returned more than 
1,000 potentially relevant results. After reviewing abstracts for each article, the study team 
identified 32 articles as relevant for a more detailed review. 

In addition to academic journal articles, the review included research reports from non- 
partisan university-affiliated and nonprofit research organizations, such as RAND, Mathe- 
matica Policy Research, the Center for Research on Education Outcomes, and the Center 
on Reinventing Public Education. This search uncovered 26 reports for review. 

Across all sources the study team identified 58 articles and reports for final review. The 
systematic review of each article or report focused on identifying primary research ques- 
tions, methodology employed, and main findings—including factors influencing student 
achievement, if applicable. In addition, the review included three articles and reports that 
did not focus explicitly on influences on student achievement but that the study team 
deemed relevant because of a focus on charter schools in New Orleans. 

Overview of study characteristics. The methodologies applied in the 58 studies reviewed 
are summarized in figure Bl. A large majority of the studies (49 of 58) used some kind 
of quantitative method to explore relationships between charter school variables (such as 
extended instruction time and size of school) and student achievement. Potential indica- 
tors were not randomly assigned to treatment or control groups, even in the studies identi¬ 
fied as experimental in this review, as the condition of random assignment was to a charter 
school or not. Thus, the findings from the literature on the potential indicators of charter 
school effectiveness should be viewed as correlational and not necessarily causal. 

Twenty-six studies employed nonexperimental quantitative methods. Twenty-three 
studies employed either quasi-experimental methods (15) or experimental methods (8). 
For the experimental studies, lottery systems to control entry into oversubscribed charter 
schools provided researchers with natural randomized treatment and control groups. A 
common quasi-experimental method involved matching charter school students (the 
treatment group) with otherwise similar traditional public school students (the control 
group). Researchers compared students across the two groups who are similar in age, 
socioeconomic status, race/ethnicity, gender, and other background characteristics. An 
additional 9 studies used other methods (qualitative, meta-analysis, literature review, and 
nonempirical). 
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Figure Bl. The most common methodology applied in the 58 studies reviewed was 
nonexperimental quantitative analysis 

Number of studies 



Nonexperimental Quasi-experimental Experimental Other methods: qualitative, 

quantitative methods meta-analysis, literature review, 

nonempirical 


Source: Authors' compilation from literature review. 


A majority of the 58 studies examined only the effect of attending charter schools on 
student achievement growth, in some cases also exploring student characteristics that 
might moderate that effect. Only 23 of the reviewed studies identified potential indicators 
of charter school effectiveness. It was from those 23 studies that the potential indicators 
for this study were drawn. Because only 5 of the 23 studies identifying potential indicators 
were experimental, factors reported in any of the studies regardless of their methodological 
rigor were included in the potential indicators list. 

Potential indicators from the studies. The literature review identified 23 potential indi¬ 
cators of charter school student achievement. Three potential indicators were excluded 
from the analyses in this study because either too few or too many charter schools had 
that feature: availability of tutoring, Knowledge is Power Program (KIPP) school model, 
and for-profit or nonprofit status. Eight potential indicators identified by prior research 
were excluded from this study because reliable measures were unavailable: administrative 
structure of the charter management organization, academic press (universally high expec¬ 
tations for student achievement), extent of instructional innovation, principal leadership, 
parent involvement, disciplinary policy, amount and difficulty of homework, and use of 
formative assessment. The 12 remaining potential indicators included in one or more of 
the analyses in this study were: 

• Authorizer type. 

• School age. 

• School size. 

• Grades served, represented in the analysis by the subindicators of number of grades 
served and inclusion of kindergarten as an entry grade. 

• Extended learning day. 

• Extended school year. 

• Percentage of teachers traditionally certified. 
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• Percentage of teachers with a graduate degree. 

• Teacher experience in years. 

• Teacher quality, represented in the analysis by the percentage of teachers rated 
highly effective. 

• Class size, represented in the analysis by student-teacher ratio. 

• Availability of student support staff (such as counselors, speech therapists, and 
occupational therapists). 

Data 

The analysis of the indicators of charter school effectiveness drew on the following data 
sources (as indicated in table Bl): 

• Data available on the Louisiana Department of Education (LDE) website Louisi¬ 
ana Believes (https://www.louisianabelieves.com). 

o The LDE’s student accountability testing database, 
o The LDE’s Student Information System database, 
o The LDE’s Profile of Educational Professionals database, 
o The LDE’s School Report Cards. 

• National Center for Education Statistics Common Core of Data (U.S. Depart¬ 
ment of Education, 2013). 

• New Orleans Parents’ Guide to Public Schools (New Orleans Parents’ Guide, 2013). 3 

All data for the main analysis were for the 2012/13 school year and were measured at or 
aggregated to the school level. Data on student achievement growth also were obtained for 
the 2013/14 school year to test the robustness of the findings of the main analysis. 

The LDE calculated school-level value-added measures (VAM) using a student-level model 
and then aggregated the results by individual school campus (there were 75 chartered 
schools in New Orleans in 2012/13, supporting 92 different campuses). Student perfor¬ 
mance on the 2013 outcome test in a given domain was estimated as a function of student 
performance on the prior-year test in that same domain as well as student background 
characteristics such as race/ethnicity, gender, low-income status (identified by eligibility 
for the federal school lunch program), English learner status, and special education status. 
The student-level residuals from the LDE regression were then converted to ^-scores by 
student grade level and grade-promotion history to adjust for higher learning trajectories 
for younger students and students who were never retained in grade. 4 Finally, the ^-scores 
for student-level residuals were aggregated to the school level. Achievement growth esti¬ 
mated in this way isolates variance in student learning that might reasonably be attributed 
to the school the student currently attends and to its various features (Chetty, Friedman, 
& Rockoff, 2012; Kane, McCaffrey, Miller, & Staiger, 2013). 

Achievement in Louisiana was assessed using two state-administered standardized tests, 
the Integrated Louisiana Educational Assessment Program (iLEAP) and the Louisiana 
Educational Assessment Program (LEAP). The iLEAP, which is administered in grades 3, 
5, 6, and 7, combines a criterion-based component and the normative Iowa Test of Basic 
Skills. The LEAP test is a high-stakes, criterion-based assessment administered in grades 4 
and 8. The criterion-based elements of both tests are based on Louisiana state standards. 
According to the LDE testing officers, the LEAP and iLEAP are similar enough in content 
and scoring to generate valid school-level VAM scores even if most individual student 
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Table Bl. Variables included in the current study of potential indicators of the effectiveness of New 

Orleans charter schools, 2012/13 


Variable 

Description 

Source 

Organizational 

Authorized by Recovery School 
District 

Indicator for whether the Recovery School District 3 was the official 
authorizer of the charter school 

Louisiana Believes 
website (https://www. 
louisianabelieves.com) 

School age 

Total years from the year of the school's founding, in any sector, until 

2013 

U.S. Department of 
Education (2013) 

School size in number of 

students 

Student enrollment on the fall 2012 enrollment count day 

U.S. Department of 
Education (2013) 

Grades served 

Number of grades 

Calculated from the reported grade range of the school 

U.S. Department of 
Education (2013) 

Includes prekindergarten 

Indicator variable calculated from the reported grade range of the school 
used exclusively as a control variable in the analysis 

U.S. Department of 
Education (2013) 

Includes kindergarten 

Indicator variable calculated from the reported grade range of the school 

U.S. Department of 
Education (2013) 

Operational 

Offers extended learning day 

Indicator of whether the school had a formal school day longer than the 

conventional seven hours 

New Orleans Parents’ 

Guide (2013) 

Offers extended school year 

Indicator of whether the school operated for more days than the state 
required 

New Orleans Parents’ 

Guide (2013) 

Instructional 

Percentage of teachers 
traditionally certified 

Proportion of the teaching staff with formal certification 

Louisiana Department 

of Education 

Percentage of teachers with a 
graduate degree 

Proportion of the teaching staff with a master's or doctorate in any field 

Louisiana Department 

of Education 

Teacher experience in years 

Average years of experience for the teaching staff 

Louisiana Department 

of Education 

Percentage of teachers rated 
highly effective 

Proportion of teachers rated highly effective at educational practice based 
on state's observational rating system 

Louisiana Department 

of Education 

Student-teacher ratio 

Total student enrollment divided by total number of teachers, both 

instructional and noninstructional 

Louisiana Department 

of Education 

Student support staff index 

Count of each of the following supports offered at the school: speech 
therapy, occupational therapy, adapted physical education, mental health 
coordinator, on-site counseling, school psychologist, mentoring, and peer 
supports 

Louisiana Department 

of Education 

Outcome 

English language arts VAM 

Average student value-added after prior-year achievement and student 
background characteristics expected to affect learning are controlled for 

Louisiana Department 

of Education 

Math VAM 

Average student value-added after prior-year achievement and student 
background characteristics expected to affect learning are controlled for 

Louisiana Department 

of Education 

Science VAM 

Average student value-added after prior-year achievement and student 
background characteristics expected to affect learning are controlled for 

Louisiana Department 

of Education 

VAM is value-added measure of school-level student achievement growth. 


a. A special school district run by the Louisiana Department of Education that intervenes in the management of chronically low- 
performing schools. 

Source: Authors’ compilation. 
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VAM estimates use iLEAP as a prior-year control and LEAP as the outcome or vice versa. 
This study excluded social studies as an outcome measure because the research literature 
supporting the identification of potential indicators was based on studies that used English 
language arts, math, or science as student outcomes. 

Before the study analysis, the study team determined that potential indicators would be 
excluded from the ordinary least squares (OLS) estimations if 90 percent or more of the 
observations had the same value for that variable. Any variable that varied so little across 
cases would be highly unlikely to explain variation in schooHevel VAM scores, which 
varied greatly and continuously across observations. Such a variable likely would have 
unacceptably high multicollinearity with other explanatory variables in the regression 
model. Two potential indicators were excluded from the analysis on this basis: providing 
free transportation to students (92 percent of the observations contained the common 
value of 1) and providing tutoring (90 percent of the observations contained the common 
value of 1). 

Sampling 

This study included two samples. Sample 1 was designed to capture the profile of all the 
public charter schools operating in New Orleans in 2012/13. All 92 charter school campus- 
es with a unique LDE site code and data indicating that they operated during the 2012/13 
school year were included. The fact that the exploratory analysis of potential indicators 
was limited to charter schools in New Orleans holds constant all city-specific factors that 
otherwise might confound the relationships in a potential indicator study with a larger 
geographic scope. 

Sample 2 was designed to inform the research question on the potential indicators of 
charter school effectiveness. Because school-level VAM scores were the measure of charter 
school effectiveness, 36 charter campuses that lacked VAM scores for 2013 for all three of 
the domains of English language arts, math, and science were excluded from this analytic 
sample. Six 2012/13 New Orleans charter schools with VAM scores were open only to 
students who passed an admissions test. Since the ability of such schools to control the 
quality of the students they admit could confound any relationship between other school 
characteristics and school effectiveness, those schools were excluded from the analytic 
sample. The 50 school campuses that remained in sample 2 represented the complete set 
of open-enrollment charter schools in New Orleans in 2012/13 with VAM outcome data 
for at least one of the three subjects. VAM scores were available in all three subjects for 47 
public charter schools in the sample. Those schools contributed to the exploratory results 
for English language arts, math, and science. VAM scores were available in English lan¬ 
guage arts and math but not science for one school, for math and science but not English 
language arts for another school, and for English language arts and science but not math 
for a third school. Thus, three schools informed the exploration of potential indicators for 
two of the subjects but not the third one. 

Language immersion charter schools were included in sample 2. Those schools require 
some familiarity with a foreign language as a minimal condition of admission. This lan¬ 
guage requirement differs in two ways from the admission-test requirements of the six 
charter schools excluded from sample 2. First, basic language familiarity is a less rigor¬ 
ous screen on student ability than the requirement to achieve a high overall score on a 
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general academic ability test. Second, test-admission charter schools are selecting students 
on initial achievement in the areas that generate the VAM scores that are the dependent 
variables in the analysis. That makes them fundamentally different from language immer- 
sion schools, conceptually and in terms of the exploratory statistical analysis. For example, 
the VAM scores have less variance in the test-admission charter schools than in the other 
charter schools, including the language immersion schools. Including both groups in the 
same statistical regression analysis would produce heteroskedastic error terms, thus vio¬ 
lating a necessary condition of regression that “the variance of the disturbance is con¬ 
stant for all observations” (Kmenta, 1986, p. 269). Heteroskedasticity reduces the efficiency 
of regression estimations, which is a serious concern for studies with small numbers of 
observations such as this one. For these reasons, language immersion charter schools were 
included in the sample while general test-admission charter schools were excluded. 


Descriptive statistics for sample 2 are shown in table B2. Student counts and composition 
for each analytic sample are in table B3. 


Table B2. Descriptive statistics for analytic sample of open-enrollment charter 
schools in New Orleans with at least some value-added measure data in 2012/13 

Statistic 

Mean 

Standard 

deviation 

Minimum 

Maximum 

Number 
of school 
campuses 

Organizational 

Authorized by Recovery School District 3 (percent) 

84.0 

na 

na 

na 

50 

School age (years) 

7.3 

2.6 

3.0 

14.0 

50 

School size (number of students) 

498.6 

167.4 

130.0 

938.0 

49 

Grades served 

Number of grades 11 

8.5 

2.0 

4.0 

14.0 

50 

Includes prekindergarten (percent) 

52.0 

na 

na 

na 

50 

Includes kindergarten (percent) 

88.0 

na 

na 

na 

50 

Includes grade 6 (percent) 

84.0 

na 

na 

na 

50 

Includes grade 12 (percent) 

6.0 

na 

na 

na 

50 

Female students (percent) 

47.7 

8.2 

0.0 

58.8 

46 

Students from low-income households (percent) 

91.4 

10.7 

51.0 

98.0 

49 

Black students (percent) 

93.7 

12.6 

47.7 

100.0 

46 

Hispanic students (percent) 

2.9 

8.1 

0.0 

51.4 

46 

English learner students (percent) 

1.5 

5.3 

0.0 

35.6 

46 

Students in special education (percent) 

10.6 

3.9 

2.5 

21.0 

49 

Operational 

Schools providing transportation (percent) 

92.0 

na 

na 

na 

50 

Schools offering extended learning day (percent) 

74.0 

na 

na 

na 

50 

Schools offering extended school year (percent) 

56.0 

na 

na 

na 

50 

Instructional 

Teachers traditionally certified (percent) 

77.4 

15.3 

31.0 

98.0 

46 

Teachers with a graduate degree (percent) 

24.5 

13.7 

0.0 

67.0 

46 

Teacher experience (years) 

7.4 

4.3 

1.2 

19.3 

50 

Teachers rated highly effective (percent) 

14.3 

22.5 

0.0 

83.0 

40 

Student-teacher ratio 

15.0 

2.6 

7.0 

21.0 

48 

Student support staff index 0 

1.8 

1.3 

0.0 

8.0 

50 

Schools providing tutoring (percent) 

90.0 

na 

na 

na 

50 

(continued) 
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Table B2. Descriptive statistics for analytic sample of open-enrollment charter 
schools in New Orleans with at least some value-added measure data in 2012/13 

(continued) 


Statistic 

Mean 

Standard 

deviation 

Minimum 

Maximum 

Number 
of school 
campuses 

Outcome 

English language arts VAM d 

2.8 

8.1 

-14.3 

29.8 

49 

Math VAM d 

7.5 

13.7 

-29.2 

44.2 

49 

Science VAM d 

1.9 

7.3 

-18.0 

17.0 

49 


na means that the descriptive statistic was not meaningful for the variable because it takes only a value of 
1 or 0; VAM is value-added measure of student achievement growth. 

Note: The highlighted variables were included in the statistical models for the potential indicator analysis. Of 
the 50 school campuses that had value-added measure scores in one or more subject area, 47 had them for 
all three domains and 3 had them for two domains. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 
of chronically low-performing schools. 

b. Grade configurations varied, but every school included at least one of the indicator grades 4-8 and most 
included two of them (for example, K and 6, 6 and 12). 

c. Count of each of the student supports offered at a school. 

d. Expressed as scaled score units, which account for different rates of learning in various grades. 

Source: Authors' calculations based on school-level data provided by the Louisiana Department of Education 
and New Orleans Parents’ Guide (2013). 


Table B3. Size and composition of study samples of New Orleans charter schools in 
2012/13 


Sample 

Focus 

Possible 

grades 

Number of 
schools 

Number of 
students 

1 

Descriptive statistics (see appendix C) 

K-12 

92 

46,064 

2 

Potential indicators of effectiveness 

4-8 

50 

12,560 


Note: The number of students informing the sample used to generate the value-added measure scores was 
estimated by multiplying the average number of students per grade at a school by the number of grades 
served in the grade 4-8 range. The number of students informing the calculation of other school-level vari¬ 
ables used in the study varied and was higher to the extent that the school included grades outside of the 
testing window of 4-8. 

Source: Authors' calculations based on school-level data provided by the Louisiana Department of Education. 


Methodology 

The research question for this study was addressed using OLS statistical regression analysis 
on schooHevel data. The purpose of OLS is to identify the pattern of association between 
a set of explanatory variables and a single dependent variable that minimizes the sum of 
the squared errors from the calculation. 

The following equation describes the OLS regression approach used here: 

VAM sk = B 0 + + e k 

where VAM jk is the 2012/13 VAM score in subject s for school k was estimated as a func- 
tion of a constant B Q and various BjS that measure the association between j indicator 
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variables measured at schooUevel k and VAM (; . The expression e k is the residual error 
after factoring in the extent to which the constant and the indicator variables explain the 
levels of VAM jk . The BjS indicate the direction of the relationship—whether the specific 
potential indicator variable is associated with higher or lower levels of VAM—as well as 
the size of the association. 

It was not necessary to control for the effects of student demographic characteristics in 
the statistical models. Student demographic characteristics, such as race/ethnicity, English 
learner status, and special education status, were all factored into the LDE’s calculation 
of schooUevel VAM. Student achievement in the previous year, 2011/12, also was a key 
element of the VAM calculations and therefore did not need to be controlled for in the 
regression models. 

An average of 3.2 percent of the data for the 12 potential indicator variables in the sta- 
tistical model was missing. Missing baseline data were multiple-imputed to preserve the 
size and representativeness of the achievement samples (King, Honaker, Joseph, & Scheve, 
2001; Puma, Olsen, Bell, & Price, 2009). None of the schooUevel VAM dependent vari- 
ables were imputed. 

A supplemental analysis using descriptive statistics was also performed comparing the 
features of the New Orleans charter school population with all Louisiana public schools 
(see appendix C). SchooUevel mean values on VAM scores and potential indicators of 
charter school effectiveness were calculated for all charter school campuses in opera- 
tion during 2012/13. Those New Orleans charter population means were compared with 
the school-level averages across all of Louisiana. Differences were calculated as the New 
Orleans charter average minus the statewide school average, and those differences were 
divided by the statewide school average to generate a percentage. 

Comparable statewide data were obtained for only half of the 26 school features for which 
descriptive information was available in sample 1 of all New Orleans public charter school 
campuses. State data privacy laws prevent the LDE from publishing the school-level state 
averages for school-level VAM scores for English language arts, math, or science as well 
as the school-level state averages for teacher years of experience, teacher certification 
percentage, and teacher graduate degree percentage. The LDE was able to provide these 
school-level data for this study with the restriction that the report would not present any 
aggregate state-level calculation of school-level VAM or those teacher characteristics. The 
measures of whether a New Orleans charter school offered tutoring, free transportation, 
extended learning day, and extended school year were taken from the New Orleans Parents’ 
Guide to Public Schools (New Orleans Parents’ Guide, 2013). Comparable information was 
not available statewide. The school-level state average for the percentage of English learner 
students was published as “less than 5 percent,” a description too vague to be used in the 
comparison with New Orleans charter schools. Finally, the indicator variable for a school 
being authorized by the Recovery School District distinguishes public charter schools 
within New Orleans but not public schools outside the city. 

This study is observational and exploratory, as it lacks the design elements required to 
identify causal relationships. It is possible that the associations identified in the analysis 
between potential indicators and student achievement growth are spurious, that is, are 
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caused by factors outside of the analysis, or endogenous, with greater effectiveness causing 
the school to adopt the indicator feature. 

Sensitivity tests 

Two sensitivity tests were run on the results from the potential indicator statistical anal- 
ysis. First, the analysis was rerun on the data using listwise deletion instead of multiple 
imputation to deal with observations with missing data. The results did not change sub- 
stantively. Second, the analysis was rerun on 2013/14 school-level VAM data provided by 
the LDE. None of the potential indicators of charter school effectiveness as measured in 
2012/13 were associated with variation in 2013/14 English language arts, math, or science 
VAM. 
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Appendix C. Characteristics of New Orleans 
charter schools and Louisiana public schools 


This appendix describes the features of the New Orleans charter school population and 
then compares them with all Louisiana public schools. SchooUevel mean values on val¬ 
ue-added measures (VAM) and potential indicators of charter school effectiveness were 
calculated for all 92 New Orleans charter school campuses in operation during 2012/13. 
The averages for the New Orleans charter population were compared with Louisiana 
school-level averages. Differences were calculated as the New Orleans charter school 
average minus the statewide school average divided by the statewide school average to 
generate a percentage difference between New Orleans charter schools and all Louisiana 
public schools relative to the average value for all schools. Significance tests could not be 
conducted on the population differences because data on the population variance for the 
census of Louisiana public schools were not available. Any apparent differences between 
the populations of New Orleans public charter schools and all Louisiana public schools 
could be due to the New Orleans schools being urban, the charter schools being public 
schools, some combination of the two, or other unknown factors. 

Although data challenges limited the number of school features the study could examine 
(see appendix A), many salient features were included (table Cl). The average age of New 
Orleans charter schools was 7 years, with the youngest having operated for 3 years and 
the oldest for 14 years. Enrollment averaged 501 students and ranged from 119 to 1,697 
students. The average number of grades served was 8, with 29 percent including grade 
12. The Recovery School District authorized 78 percent of New Orleans charter schools. 
The proportion of students from low-income households (as indicated by eligibility for the 
federal school lunch program) averaged 91 percent, the proportion of Black students aver¬ 
aged 91 percent, and the proportion of students in special education averaged 11 percent. 


Table Cl. Descriptive statistics for the census of 2012/13 New Orleans public 
charter school campuses, 2012/13 

Statistic 

Mean 

Standard 

deviation 

Minimum 

Maximum 

Number 
of school 
campuses 

Organizational 

Authorized by Recovery School District 3 (percent) 

77.8 

na 

na 

na 

90 

School age (years) 

7.4 

2.3 

3.0 

14.0 

86 

School size (number of students) 

500.7 

240.6 

119.0 

1,697.0 

81 

Grades served 

Number of grades 11 

7.5 

2.8 

1.0 

14.0 

89 

Includes prekindergarten (percent) 

41.6 

na 

na 

na 

89 

Includes kindergarten (percent) 

65.2 

na 

na 

na 

89 

Includes grade 6 (percent) 

65.2 

na 

na 

na 

89 

Includes grade 12 (percent) 

29.2 

na 

na 

na 

89 

Female students (percent) 

47.6 

8.3 

0.0 

67.3 

61 

Students from low-income households (percent) 

90.6 

10.3 

51.0 

99.7 

78 

Black students (percent) 

90.9 

17.7 

18.3 

100.0 

61 

Hispanic students (percent) 

2.9 

7.2 

0.0 

51.4 

61 

English learner students (percent) 

1.7 

4.9 

0.0 

35.6 

61 

(continued) 
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Table Cl. Descriptive statistics for the census of 2012/13 New Orleans public 
charter school campuses, 2012/13 ( continued ) 

Statistic 

Mean 

Standard 

deviation 

Minimum 

Maximum 

Number 
of school 
campuses 

Students in special education (percent) 

10.8 

4.2 

2.5 

22.0 

78 

Operational 

Schools providing transportation (percent) 

95.5 

na 

na 

na 

89 

Schools offering extended learning day (percent) 

71.9 

na 

na 

na 

89 

Schools offering extended school year (percent) 

52.8 

na 

na 

na 

89 

Instructional 

Teachers traditionally certified (percent) 

79.6 

14.8 

31.0 

98.0 

58 

Teachers with a graduate degree (percent) 

24.9 

14.8 

0.0 

67.0 

58 

Teacher experience (years) 

7.7 

4.3 

1.2 

19.3 

70 

Teachers rated highly effective (percent) 

12.7 

20.1 

0.0 

83.0 

56 

Student-teacher ratio 

14.8 

2.6 

7.0 

21.0 

78 

Student support staff index 0 

1.8 

1.1 

0.0 

8.0 

89 

Schools providing tutoring (percent) 

93.3 

na 

na 

na 

89 

Outcome 

English language arts VAM d 

2.9 

7.9 

- 14.3 

29.8 

55 

Math VAM d 

7.3 

13.3 

- 29.2 

44.2 

55 

Science VAIVT 1 

1.7 

7.2 

- 18.0 

17.0 

55 


na means that the descriptive statistic was not meaningful for the variable because it only takes the value 
1 or 0; VAM is value-added measure of student achievement growth. 

Note: A total of 92 public charter school campuses operated in New Orleans in 2012/13. The number of 
school campuses with data for each characteristic varied substantially because many campuses were not 
required to report some information to the state, they were required but neglected to do so, or they reported 
the information but it was lost or corrupted in the process of data aggregation. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 
of chronically low-performing schools. 

b. Grade configurations varied, but every school included at least one of the indicator grades 4-8 and most 
included two of them (for example, K and 6, 6 and 12). 

c. Count of each of the student supports offered at a school. 

d. Expressed as scaled score units, which account for different rates of learning in various grades. 

Source: Authors' compilation from school-level data provided by the Louisiana Department of Education and 
New Orleans Parents’ Guide (2013). 


A total of 72 percent of New Orleans charter schools provided an extended learning day, and 
53 percent provided an extended school year. The proportion of teachers with traditional 
certification averaged 80 percent, with an average of 25 percent having a graduate degree. 
Teachers averaged 7.7 years of teaching experience, and 13 percent of teachers were rated 
highly effective in practice. The average student-teacher ratio was 15, ranging from 7 to 21. 

The average New Orleans charter school offered more grades than the average Louisiana school 

In 2012/13 New Orleans charter schools were similar to all public schools in the state on some 
key characteristics and distinctive on others (table C2). The two populations of schools were 
considered approximately similar on a characteristic if the difference was less than 5 percent 
and descriptively distinctive if the difference was 5 percent or more. 5 The average for New 
Orleans charter schools was similar to the statewide average for school size, percentage of 
female students, percentage of students in special education, and student-teacher ratio. 
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Table C2. Comparison of New Orleans public charter schools and state public 
schools on selected characteristics, 2012/13 

Characteristic 

New Orleans 
charter mean 

Louisiana 

mean 

Difference 

Percent 

difference 

Organizational 

School size (number of students) 

500.7 

493.3 

7.4 

1.5 

Number of grades 3 

7.5 

5.9 

1.6 

27.1 

Includes kindergarten (percent) 

65.2 

49.6 

15.6 

31.5 

Includes grade 6 (percent) 

65.2 

36.4 

28.8 

79.1 

Includes grade 12 (percent) 

29.2 

23.0 

6.2 

27.0 

Female students (percent) 

47.6 

48.7 

-1.1 

-2.3 

Low-income students (percent) 

90.6 

66.3 

24.3 

36.7 

Black students (percent) 

90.9 

45.0 

45.9 

102.0 

Hispanic students (percent) 

2.9 

4.3 

-1.5 

-32.6 

Students in special education (percent) 

10.8 

11.3 

-0.5 

-4.4 

Instructional 

Teachers rated highly effective (percent) 

12.7 

18.3 

-5.6 

-30.6 

Student-teacher ratio 

14.8 

15.2 

-0.4 

-2.6 

Student support staff index b 

1.8 

3.2 

-1.4 

-43.8 


Note: The two populations of schools were considered approximately similar on a characteristic if the differ¬ 
ence was less than 5 percent and descriptively distinctive if the difference was 5 percent or more. Descrip¬ 
tively distinctive characteristics are shown in bold. 

a. Grade configurations varied, but every school included at least one of the indicator grades 4-8 and most 
included two of them (for example, K and 6, 6 and 12). 

b. Count of each of the student supports offered at a school. 

Source: Authors' compilation from school-level data provided by the Louisiana Department of Education and 
New Orleans Parents' Guide (2013). 


New Orleans charter schools were descriptively distinctive in that they offered 27 percent 
more grades, on average, than the statewide norm (7.5 compared with 5.9). The average 
New Orleans charter school was much more likely to include kindergarten (32 percent 
more likely), grade 6 (79 percent more likely), and grade 12 (27 percent more likely) than 
the average Louisiana public school. Charter schools more often included broad grade 
ranges such as K—6, K—8, or even K-12 that cut across typical schoohbuilding transition 
grades. 

The average New Orleans charter school served more Black students and more students from low- 
income households than the average Louisiana school did 

The proportion of Black students in the average New Orleans charter school (91 percent) 
was more than twice the statewide average (45 percent), and the proportion of students 
from low-income households was more than a third higher (91 percent) than the statewide 
average (66 percent). The average proportion of Hispanic students in New Orleans charter 
schools (3 percent) was lower than the statewide average (4 percent). The average score on 
the student support staff index (count of student supports offered at a particular school) for 
New Orleans charter schools (1.8) was 1.4 points lower than the statewide average (3.2). 
On average, 12.7 percent of teachers in New Orleans charter schools were rated highly 
effective, 5.6 percentage points less than the average of 18.3 percent for all public schools 
in Louisiana. 
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Appendix D. Detailed results 


This appendix reports the results of the regression analysis of potential indicators of New 
Orleans charter school effectiveness for English language arts value-added measures (table 
Dl), math YAM (table D2), and science YAM (table D3). 


Table Dl. Association of potential indicators of effectiveness in New Orleans 
public charter schools for English language arts value-added measure, 2012/13 
(unstandardized regression coefficient estimates) 


Characteristic 

Beta 

Standard error 

p value 

Organizational 

Authorized by Recovery School District 8 

3.78 

3.50 

.29 

School age 

-0.78 

0.49 

.12 

School size in number of students 

1.48 

0.78 

.07 

Number of grades served 

-1.83 

0.95 

.06 

Includes prekindergarten 

-4.51 

2.95 

.14 

Includes kindergarten 

15.30 

4.21** 

.00 

Operational 

Offers extended learning day 

-3.11 

2.95 

.30 

Offers extended school year 

5.22 

2.33* 

.03 

Instructional 

Percentage of teachers traditionally certified 

12.20 

9.33 

.21 

Percentage of teachers with a graduate degree 

-25.55 

10.16* 

.02 

Teacher experience in years 

1.09 

0.46* 

.02 

Percentage of teachers rated highly effective 

2.87 

4.14 

.59 

Student-teacher ratio 

-0.92 

0.39* 

.03 

Student support staff index b 

-0.15 

0.83 

.86 


* Significant at p < .05; ** significant at p < .01. 

Note: Value-added measure scores are in scale score units, which account for different rates of learning in 
various grades. F-statistic = 3.10; probability > F = .00. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 
of chronically low-performing schools. 

b. Count of each of the student supports offered at a school. 

Source: Authors' calculations based on school-level data provided by the Louisiana Department of Education 
and New Orleans Parents’ Guide (2013). 
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Table D2. Association of potential indicators of effectiveness in New Orleans 
public charter schools for math value-added measure, 2012/13 (unstandardized 
regression coefficient estimates) 


Characteristic 

Beta 

Standard error 

p value 

Organizational 

Authorized by Recovery School District 8 

4.65 

7.11 

.52 

School age 

-0.78 

1.06 

.47 

School size in number of students 

2.08 

1.57 

.20 

Number of grades served 

-2.21 

2.03 

.29 

Includes prekindergarten 

-7.06 

6.03 

.25 

Includes kindergarten 

22.59 

9.04* 

.02 

Operational 

Offers extended learning day 

-0.95 

6.25 

.88 

Offers extended school year 

6.45 

4.94 

.20 

Instructional 

Percentage of teachers traditionally certified 

11.15 

20.65 

.60 

Percentage of teachers with a graduate degree 

-33.83 

22.57 

.15 

Teacher experience in years 

0.98 

1.00 

.34 

Percentage of teachers rated highly effective 

1.15 

9.31 

.90 

Student-teacher ratio 

-0.82 

0.82 

.33 

Student support staff index b 

-0.90 

1.76 

.61 


* Significant at p < .05; ** significant at p < .01 

Note: Value-added measure scores are in scale score units, which account for different rates of learning in 
various grades. F-statistic = 1.09; probability > F = .41. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 
of chronically low-performing schools. 

b. Count of each of the student supports offered at a school. 

Source: Authors' calculations based on school-level data provided by the Louisiana Department of Education 
and New Orleans Parents’ Guide (2013). 
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Table D3. Association of potential indicators of effectiveness in New Orleans 
public charter schools for science value-added measure, 2012/13 (unstandardized 
regression coefficient estimates) 


Characteristic 

Beta 

Standard error 

p value 

Organizational 

Authorized by Recovery School District 8 

2.85 

4.25 

.51 

School age 

0.13 

0.59 

.83 

School size in number of students 

0.08 

0.86 

.92 

Number of grades served 

0.59 

1.12 

.60 

Includes prekindergarten 

-2.12 

3.37 

.53 

Includes kindergarten 

0.31 

4.97 

.95 

Operational 

Offers extended learning day 

-2.42 

3.43 

.49 

Offers extended school year 

3.09 

2.75 

.27 

Instructional 

Percentage of teachers traditionally certified 

-0.74 

10.86 

.95 

Percentage of teachers with a graduate degree 

-4.49 

12.05 

.71 

Teacher experience in years 

0.08 

0.56 

.88 

Percentage of teachers rated highly effective 

-1.86 

4.57 

.69 

Student-teacher ratio 

-1.24 

0.45* 

.01 

Student support staff index b 

-2.00 

0.97* 

.05 


* Significant at p < .05; ** significant at p < .01 

Note: Value-added measure scores are in scale score units, which account for different rates of learning in 
various grades. F-statistic = 1.27; probability > F = .28. 

a. A special school district run by the Louisiana Department of Education that intervenes in the management 
of chronically low-performing schools. 

b. Count of each of the student supports offered at a school. 

Source: Authors' calculations based on school-level data provided by the Louisiana Department of Education 
and New Orleans Parents’ Guide (2013). 
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Notes 


1. Core alliance members represent the following institutions: Andrew H. Wilson 
Charter School, Audubon Charter School, Benjamin Franklin High School, Cypress 
Academy, Einstein Elementary Charter School, Greater New Orleans Collaborative of 
Charter Schools, Hynes Charter School, International School of Louisiana, Jefferson 
Parish School District, Lake Forest Elementary Charter School, Louisiana Depart- 
ment of Education, Lusher Charter School, Magnolia School of Excellence, Morris 
Jeff Community School, New Beginnings Schools Foundation, New Orleans Charter 
Science and Mathematics High School, School Leadership Center of Greater New 
Orleans, Tulane University, Warren Easton Charter High School, West Feliciana 
Parish Schools, and Xavier University of Louisiana. 

2. High school students in grades 9-12 are assessed by end-of-course exams that are 
administered by subject and are not comparable from year to year. As such, high 
schools that do not include grade 7 or 8 are not included in this analysis. 

3. New Orleans Parents’ Guide to Public Schools is produced annually by the New Orleans 
Parents’ Guide, a nonprofit community organization, and includes an associated 
smartphone app. Staff and volunteers of the New Orleans Parents’ Guide collect infor¬ 
mation directly from all the public schools in the city and verify those data with LDE 
prior to publishing the guide. 

4. Grade promotion history was accounted for in the conversion to a 2 -score of the depen¬ 
dent variable and not controlled for in the regression estimations themselves because 
the analysis already suffered from having a small number of degrees of freedom. 

5. Five percent was selected as the benchmark for differences being distinctive because a 
difference of that magnitude can be noticeable to the observer, whereas a difference of 
less than 5 percent tends to be imperceptible. Information about the variance around 
the mean for the statewide averages was not available, preventing the use of formal 
significance tests in judging the reliability of the differences. 
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